Siamese Convolutional Network Research Articles

Learning Regression Distribution: Information Diffusion from Template to Search for Visual Object Tracking

Article Learning Regression Distribution: Information Diffusion from Template to Search for Visual Object Tracking Shuo Hu *, Jinbo Lu, and Sien Zhou 1 School of Electrical Engineering, Yanshan University, Qinhuangdao 066000, China * Correspondence: hus@ysu.edu.cn     Received: 5 July 2023 Accepted: 19 October 2023 Published: 26 March 2024   Abstract: The general paradigm of traditional Siamese networks involves using cross-correlations to fuse features from the backbone, and this paradigm is limited by the inductive bias of the convolution kernel, resulting in the lack of global information. In this paper, we propose the Siamese learning regression distribution (SiamLRD) to address the local limitations of traditional cross-correlation operations on feature fusion and weak self-connections between features within different branches. The SiamLRD uses the cross-attention mechanism to replace cross-correlations between the features of the target region of interest and the template so as to enhance flexibility. Firstly, the original transformer structure is improved to be suitable for convolutional Siamese networks. The improved transformer architecture is then used to replace cross-correlation operations, resulting in more comprehensive feature fusion between branches. Secondly, we introduce a new decoder structure into the novel fusion strategy to enhance the correlation between classification scores and regression accuracy during decoding. Multiple benchmarks are used to test the proposed SiamLRD approach, and it is verified that the proposed approach improves the baseline with 5.8% in terms of AO and 9.7% in terms of SR0.75 on the GOT-10K dataset.

A New Siamese Heterogeneous Convolutional Neural Networks Based on Attention Mechanism and Feature Pyramid.

Accuracy and speed are the most important indexes for evaluating many object tracking algorithms. However, when constructing a deep fully convolutional neural network (CNN), the use of deep network feature tracking will cause tracking drift due to the effects of convolution padding, receptive field (RF), and overall network step size. The speed of the tracker will also decrease. This article proposes a fully convolutional siamese network object tracking algorithm that combines the attention mechanism with the feature pyramid network (FPN), and uses heterogeneous convolution kernels to reduce the amount of calculations (FLOPs) and parameters. The tracker first uses a new fully CNN to extract image features, and introduces a channel attention mechanism in the feature extraction process to improve the representation ability of convolutional features. Then use the FPN to fuse the convolutional features of high and low layers, learn the similarity of the fused features, and train the fully CNNs. Finally, the heterogeneous convolutional kernel is used to replace the standard convolution kernel to improve the speed of the algorithm, thereby making up for the efficiency loss caused by the feature pyramid model. In this article, the tracker is experimentally verified and analyzed on the VOT-2017, VOT-2018, OTB-2013, and OTB-2015 datasets. The results show that our tracker has achieved better results than the state-of-the-art trackers.

Multi-scale convolutional neural networks and saliency weight maps for infrared and visible image fusion

Image fusion is the fusion of multiple images from the same scene to produce a more informative image, and infrared and visible image fusion is an important branch of image fusion. To tackle the issues of diminished luminosity in the infrared target, inconspicuous target features, and blurred texture of the fused image after the fusion of infrared and visible images. This paper introduces a novel effective fusion framework that merges multi-scale Convolutional Neural Networks (CNN) with saliency weight maps. First, the method measures the source image features to estimate the initial saliency weight map. Then, the initial weight map is segmented and optimized using a guided filter before being further processed by CNN. Next, a trained Siamese convolutional network is used to solve the two key problems of activity measure and weight assignment. Meanwhile, a multi-layer fusion strategy is designed to effectively retain the luminance of the infrared target and the texture information in the visible background. Finally, adaptive adjustment of the fusion coefficients is achieved by employing saliency. The experimental results show that the method outperforms the state-of-the-art algorithms in terms of both subjective visual quality and objective evaluation effects.

SpeakerNet for Cross-lingual Text-Independent Speaker Verification

Biometrics provide an alternative to passwords and pins for authentication. The emergence of machine learning algorithms provides an easy and economical solution to authentication problems. The phases of speaker verification protocol are training, enrollment of speakers and evaluation of unknown voice. In this paper, we addressed text independent speaker verification using Siamese convolutional network. Siamese networks are twin networks with shared weights. Feature space can be learnt easily by training these networks even if similar observations are placed in proximity. Extracted features from Siamese then can be classified using difference or correlation measures. We have implemented a customized scoring scheme that utilizes Siamese’ capability of applying distance measures with the convolutional learning. Experiments made on cross language audios of multi-lingual speakers confirm the capability of our architecture to handle gender, age and language independent speaker verification. Moreover, our designed Siamese network, SpeakerNet, provided better results than the existing speaker verification approaches by decreasing the equal error rate to 0.02.

Data association for maneuvering targets through a combined siamese network and XGBoost model

Data association plays an important role in forming target tracks when false alarms exist. Its accuracy is key to reducing the computational burden of the combinatorial explosion problem inherent to target tracking in environments where false alarms are densely distributed. Traditional methods are usually developed according to some assumed motion models and suffer performance degradation when there is mismatch between assumed models and the actual motion, which may often be the case for maneuvering targets. To cope with this problem, we propose a data-driven method that learns the association criteria directly from data. By use of the domain knowledge and a convolutional Siamese network, features of different attributes present in the observations are extracted. Then the features are fused through the use of XGBoost. Simulation results show that the proposed method performs better than the traditional model-based methods in correlating observations of maneuvering targets and avoiding correlations of false alarms. By visualizing the decision process of the feature fusion model, it is found that the proposed method learns from the data to make comprehensive use of multiple features and does not simply set hard thresholds for the features. The computational complexity is also analyzed both theoretically and experimentally.

A Siamese Network based Writer Independent Offline Signature Verification

Signatures are frequently used for both personal authorization and verification. The signatures on numerous papers, including legal agreements and bank checks, must be verified. Offline signature verification is a type of authentication that examines the physical action of signing while measuring the characteristics of an user's handwriting. In the present paper, an offline signature verification method was developed utilising Convolutional Siamese Network of Deep Learning [4]. Convolutional Siamese networks, which are dual networks that shares parameters, can be developed to acquire a feature map. Thus, Convolutional Siamese network is employed to verify the person's signature to a input signature that is saved in the database. Research has been conducted with a dataset of signatures that includes 750 signatures from 30 different individuals. An Accuracy of 91.6% was attained by the proposed solution.

On Pseudorandomness and Deep Learning: A Case Study

Pseudorandomness is a crucial property that the designers of cryptographic primitives aim to achieve. It is also a key requirement in the calls for proposals of new primitives, as in the case of block ciphers. Therefore, the assessment of the property is an important issue to deal with. Currently, an interesting research line is the understanding of how powerful machine learning methods are in distinguishing pseudorandom objects from truly random objects. Moving along such a research line, in this paper a deep learning-based pseudorandom distinguisher is developed and trained for two well-known lightweight ciphers, Speck and Simon. Specifically, the distinguisher exploits a convolutional Siamese network for distinguishing the outputs of these ciphers from random sequences. Experiments with different instances of Speck and Simon show that the proposed distinguisher highly able to distinguish between the two types of sequences, with an average accuracy of 99.5% for Speck and 99.6% for Simon. Hence, the proposed method could significantly impact the security of these cryptographic primitives and of the applications in which they are used.

Change Detection Based on Supervised Contrastive Learning for High-Resolution Remote Sensing Imagery

Change detection (CD) is a challenging task on high-resolution bitemporal remote sensing images. Many recent studies of CD have focused on designing fully convolutional Siamese network architectures. However, most of these methods initialize their encoders by random values or an ImageNet pretrained model, without any prior for the CD task, thus limiting the performance of the CD model. In this article, the novel supervised contrastive pretraining and fine-tuning CD (SCPFCD) framework, which is made up of two cascaded stages, is presented to train a CD network based on a pretrained encoder. In the first supervised contrastive pretraining stage, the encoder of the Siamese network is asked to solve a joint pretext task introduced by the proposed CDContrast pretraining method on labeled CD data. The proposed CDContrast pretraining method includes land contrastive learning (LCL), which is based on supervised contrastive learning, and proxy CD learning. The LCL focuses on learning the spatial relationships among the land cover from bitemporal images by solving a land contrast task, while the proxy CD learning performs a proxy CD task on the top of the upsampling projector to avoid local optima for the LCL and learn features for the CD. Then, in the second fine-tuning stage, the whole Siamese network initialized with the pretrained encoder is fine-tuned to perform the CD task in an end-to-end manner. The proposed SCPFCD framework was verified with three CD datasets of high-resolution remote sensing images. The extensive experimental results consistently show that the proposed framework can effectively improve the ability to extract change information for Siamese networks.

Medical image fusion using convolutional neural network

Medical image fusion methods combine medical pictures from many morphologies to improve the accuracy and reliability of medical diagnoses, and they are becoming more significant in a variety of clinical applications. This research introduces a convolutional neural network (CNN) based medical image fusion approach to create a fused picture with good visual quality and clear structural details. To generate a weight map, the proposed technique employs a trained Siamese convolutional network to fuse the pixel activity information of source pictures. Meanwhile, the original picture is decomposed using a contrast pyramid. Source pictures are combined using distinct spatial frequency bands and a weighted fusion operator. The suggested fusion method can successfully maintain the exact structural information of source pictures and generate excellent human visual effects, according to the findings of comparison trials.

Feature distillation Siamese networks for object tracking

In recent years, Siamese-based trackers have shown remarkable improvement in visual tracking. The general trends are interested in making deeper and more complicated networks to pursue higher accuracy. However, these advances result in cumbersome trackers with respect to size and speed, which hinders the deployment of deep trackers on edge devices. Due to the tracking scenario complexity and temporal coherence, the backbone network of deep trackers emphasizes the target appearance information more. However, the appearance feature is sensitive to parameter variation, making the traditional deep model compression methods hard to compress a deep tracker. To bridge the gap between deep Siamese trackers and practical use, we propose a new feature distillation algorithm suitable for deep trackers in this paper. Firstly, motivated by the concept of divide-and-conquer, we formulate the feature distillation into a stepwise distillation problem and perform distillation on each minimum unit to relieve the hard-to-distill problem of appearance feature. Secondly, we reconstruct the student model into a combination of convolution kernels and a point-wise convolutional layer, which enables the student model to inherit all the parameters of the teacher model during initialization. Finally, we propose a 3-step warm-up training strategy to address the student model’s degradation and structural adaptation problems during training. Extensive experiments on eight benchmarks demonstrate that our proposed method compresses the fully convolutional Siamese Networks (SiamFC) and its variant Siamd and achieves leading tracking performance with only 2.1 MB model size while running at 225 fps.

SNRMPACDC: computational model focused on Siamese network and random matrix projection for anticancer synergistic drug combination prediction.

Synergistic drug combinations can improve the therapeutic effect and reduce the drug dosage to avoid toxicity. In previous years, an in vitro approach was utilized to screen synergistic drug combinations. However, the in vitro method is time-consuming and expensive. With the rapid growth of high-throughput data, computational methods are becoming efficient tools to predict potential synergistic drug combinations. Considering the limitations of the previous computational methods, we developed a new model named Siamese Network and Random Matrix Projection for AntiCancer Drug Combination prediction (SNRMPACDC). Firstly, the Siamese convolutional network and random matrix projection were used to process the features of the two drugs into drug combination features. Then, the features of the cancer cell line were processed through the convolutional network. Finally, the processed features were integrated and input into the multi-layer perceptron network to get the predicted score. Compared with the traditional method of splicing drug features into drug combination features, SNRMPACDC improved the interpretability of drug combination features to a certain extent. In addition, the introduction of convolutional networks can better extract the potential information in the features. SNRMPACDC achieved the root mean-squared error of 15.01 and the Pearson correlation coefficient of 0.75 in 5-fold cross-validation of regression prediction for response data. In addition, SNRMPACDC achieved the AUC of 0.91 ± 0.03 and the AUPR of 0.62 ± 0.05 in 5-fold cross-validation of classification prediction of synergistic or not. These results are almost better than all the previous models. SNRMPACDC would be an effective approach to infer potential anticancer synergistic drug combinations.

A Region-Based Feature Fusion Network for VHR Image Change Detection

Deep learning (DL)-based architectures have shown a strong capacity to identify changes. However, existing change detection (CD) networks still suffer from limited applicability when it comes to multi-scale targets and spatially misaligned objects. For the sake of tackling the above problems, a region-based feature fusion network (RFNet) for CD of very high spatial resolution (VHR) remote sensing images is proposed. RFNet uses a fully convolutional Siamese network backbone where a multi-stage feature interaction module (MFIM) is embedded in the dual encoder and a series of region-based feature fusion modules (RFFMs) is used to generate change information. The MFIM fuses features in different stages to enhance the interaction of multi-scale information and help the network better distinguish complex ground objects. The RFFM is built based on region similarity (RSIM), which measures the similarity of bitemporal features with neighborhoods. The RFFM can reduce the impact of spatially offset bitemporal targets and accurately identify changes in bitemporal images. We also design a deep supervise strategy by directly introducing RSIM into loss calculation and shortening the error propagation distance. We validate RFNet with two popular CD datasets: the SECOND dataset and the WHU dataset. The qualitative and quantitative comparison results demonstrate the high capacity and strong robustness of RFNet. We also conduct robustness experiments and the results demonstrate that RFNet can deal with spatially shifted bitemporal images.

A deep learning method for automatic evaluation of diagnostic information from multi-stained histopathological images

Manual screening of large-scale histopathological images is an extremely time-consuming, laborious and subjective procedure. Accurate evaluation of diagnostic information from multi-color stained images requires expertise due to the complex nature of histopathology and the lack of quantifiable measurement. In this work, a novel deep learning method is developed based on a convolutional siamese network, in which the information quantification task is transformed into a similarity assessment between lesion and non-lesion patterns on histopathological images. The subtle changes underlying the microstructure of tissue biopsies can be captured through an optimization of training loss within a low-to-high-level feature space. A new information score is introduced to quantify the abnormality in tissue appearance and stain pattern. Experiments on 3 independent data cohorts including 5 types of color-stained images demonstrate that our method can achieve promising performance compared with state-of-the-art methods. Results show that the proposed information score can serve as an effective measure to evaluate the importance of multi-stained images, and ultimately facilitate automatic diagnosis for clinical multi-stained histopathology.

Improved Fully Convolutional Siamese Networks for Visual Object Tracking Based on Response Behaviour Analysis.

Siamese networks have recently attracted significant attention in the visual tracking community due to their balanced accuracy and speed. However, as a result of the non-update of the appearance model and the changing appearance of the target, the problem of tracking drift is a regular occurrence, particularly in background clutter scenarios. As a means of addressing this problem, this paper proposes an improved fully convolutional Siamese tracker that is based on response behaviour analysis (SiamFC-RBA). Firstly, the response map of the SiamFC is normalised to an 8-bit grey image, and the isohypse contours that represent the candidate target region are generated through thresholding. Secondly, the dynamic behaviour of the contours is analysed in order to check if there are distractors approaching the tracked target. Finally, a peak switching strategy is used as a means of determining the real tracking position of all candidates. Extensive experiments conducted on visual tracking benchmarks, including OTB100, GOT-10k and LaSOT, demonstrated that the proposed tracker outperformed the compared trackers such as DaSiamRPN, SiamRPN, SiamFC, CSK, CFNet and Staple and achieved state-of-the-art performance. In addition, the response behaviour analysis module was embedded into DiMP, with the experimental results showing the performance of the tracker to be improved through the use of the proposed architecture.

Antiocclusion Visual Tracking Algorithm Combining Fully Convolutional Siamese Network and Correlation Filtering.

Machine learning only uses single-channel grayscale features to model the target, and the filter solution process is relatively simple. When the target has a large change relative to the initial frame, the tracking effect is poor. When there is the same kind of target interference in the target search area, the tracking results will be poor. The tracking algorithm based on the fully convolutional Siamese network can solve these problems. By learning the similarity measurement function, the similarity between the template and the target search area is evaluated, and the target area is found according to the similarity. It adopts offline pre-training and does not update online for tracking, which has a faster tracking speed. According to this study, (1) considering the accuracy and speed, the target tracking algorithm based on correlation filtering performs well. A sample adaptive update model is introduced to eliminate unreliable samples, which effectively enhances the reliability of training samples. With simultaneous changes in illumination and scale, fast motion and in-plane rotation IPR can still be maintained. (2) Determined by calculating the Hessian matrix, in the Struck function, Bike3 parameter adjustment can achieve fast tracking, and Boat5 ensures that the system stability is maintained in the presence of interference factors. The position of the highest scoring point in the fine similarity score map of the same size as the search image is obtained by bicubic interpolation as the target position. (3) The parallax discontinuity caused by the object boundary cannot be directly processed as a smooth continuous parallax. The MeanShift vector obtained by calculating the target template feature and the feature to be searched can increase the accuracy by 53.1%, reduce the robustness by 31.8%, and reduce the error by 28.6% in the SiamVGG algorithm.

PENet: Continuous-Valued Pulmonary Edema Severity Prediction On Chest X-ray Using Siamese Convolutional Networks.

For physicians to make rapid clinical decisions for patients with congestive heart failure, the assessment of pulmonary edema severity in chest radiographs is vital. Although deep learning has shown promise in detecting the presence or absence or discrete grades of severity, of such edema, prediction of continuous-valued severity yet remains a challenge. Here, we propose PENet: Siamese convolutional neural networks to assess the continuous spectrum of severity of lung edema from chest radiographs. We present different modes of implementing this network and demonstrate that our best model outperforms that of earlier work (mean AUC of 0.91 over 0.87), while using only 1/16-th the dimension of input images and 1/69-th the size of training data, thus also saving expensive computation.

Transforming OMIC features for classification using siamese convolutional networks.

Modern biotechnologies have generated huge amount of OMIC data, among which transcriptomes and methylomes are two major OMIC types. Transcriptomes measure the expression levels of all the transcripts while methylomes depict the cytosine methylation levels across a genome. Both OMIC data types could be generated by array or sequencing. And some studies deliver many more features (the number of features is denoted as [Formula: see text]) for a sample than the number [Formula: see text] of samples in a cohort, which induce the "large [Formula: see text] small [Formula: see text]" paradigm. This study focused on the classification problem about OMIC with "large [Formula: see text] small [Formula: see text]" paradigm. A Siamese convolutional network was utilized to transform the OMIC features into a new space with minimized intra-class distances and maximized inter-class distances between the samples. The proposed feature engineering algorithm SiaCo was comprehensively evaluated using both transcriptome and methylome datasets. The experimental data showed that SiaCo generated SiaCo features with improved classification accuracies for binary classification problems, and achieved improvements on the independent test dataset. The individual SiaCo features did not show better inter-class discrimination powers than the original OMIC features. This may be due to that the Siamese convolutional network optimized the collective performances of the SiaCo features, instead of the individual feature's discrimination power. The inherent transformation nature of the Siamese twin network also makes the SiaCo features lack of interpretability. The source code of SiaCo is freely available at http://www.healthinformaticslab.org/supp/resources.php.

Semantic feature-constrained multitask siamese network for building change detection in high-spatial-resolution remote sensing imagery

In the field of remote sensing applications, semantic change detection (SCD) simultaneously identifies changed areas and their change types by jointly conducting bitemporal image classification and change detection. It facilitates change reasoning and provides more application value than binary change detection (BCD), which offers only a binary map of the changed/unchanged areas. In this study, we propose a multitask Siamese network, named the semantic feature-constrained change detection (SFCCD) network, for building change detection in bitemporal high-spatial-resolution (HSR) images. SFCCD conducts feature extraction, semantic segmentation and change detection simultaneously, where change detection and semantic segmentation are the main and auxiliary tasks, respectively. For the segmentation task, ResNet50 is used to conduct image feature extraction, and the extracted semantic features are provided to execute the change detection task via a series of jump connections. For the change detection task, a global channel attention (GCA) module and a multiscale feature fusion (MSFF) module are designed, where high-level features offer training guidance to the low-level feature maps, and multiscale features are fused with multiple convolutions that possess different receptive fields. In bitemporal HSR images with different view angles, high-rise buildings have different directional height displacements, which generally cause serious false alarms for common change detection methods. However, known public building change detection datasets often lack buildings with height displacement. We thus create the Nanjing Dataset (NJDS) and design the aforementioned network structures and modules to target this issue. Experiments for method validation and comparison are conducted on the NJDS and two additional public datasets, i.e., the WHU Building Dataset (WBDS) and Google Dataset (GDS). Ablation experiments on the NJDS show that the joint utilization of the GCA and MSFF modules performs better than several classic modules, including atrous spatial pyramid pooling (ASPP), efficient spatial pyramid (ESP), channel attention block (CAB) and global attention upsampling (GAU) modules, in dealing with building height displacement. Furthermore, SFCCD achieves higher accuracy in terms of the OA, recall, F1-score and mIoU measures than several state-of-the-art change detection methods, including deeply supervised image fusion network (DSIFN), the dual-task constrained deep Siamese convolutional network (DTCDSCN), and multitask U-Net (MTU-Net).

Multiscale anti-deformation network for target tracking in UAV aerial videos

With the popularity of unmanned aerial vehicle (UAV) technology, remote sensing target tracking in aerial videos from UAVs has drawn much attention. However, various problems occur, such as obvious target scale changes and frequent target shape updates in UAV aerial videos, and typical tracking algorithms have difficulties in solving these challenges. Therefore, we propose a remote sensing target tracking method for UAV videos based on a multiscale antideformation network (MSADN). This method uses the fully convolutional Siamese network (SiameseFC) as its basic architecture. First, in the target feature extraction stage, we use a receptive field block module to change the single convolutional layer of the network. And the multibranch structure can improve the algorithm’s adaptability to target scale changes. Then, in the tracking stage, to solve the problem of the lack of template updates, we integrate a template dynamic update module into the SiameseFC architecture. This module uses long short-term memory to generate control signals to dynamically update target shape information, and it will improve the algorithm’s ability to cope with frequent target shape updates. Finally, compared to state-of-the-art trackers, experimental results show that the MSADN algorithm can obtain better performance (75.9% Prec, 56.4% Succ) while ensuring higher efficiency (60 fps).

Enhancement: SiamFC Tracker Algorithm Performance Based on Convolutional Hyperparameters Optimization and Low Pass Filter

Over the past few decades, convolutional neural networks (CNNs) have achieved outstanding results in addressing a broad scope of computer vision problems. Despite these improvements, fully convolutional Siamese neural networks (FCSNN) still hardly adapt to complex scenes, such as appearance change, scale change, similar objects interference, etc. The present study focuses on an enhanced FCSNN based on convolutional block hyperparameters optimization, a new activation function (ModReLU) and Gaussian low pass filter. The optimization of hyperparameters is an important task, as it has a crucial ascendancy on the tracking process performance, especially when it comes to the initialization of weights and bias. They have to work efficiently with the following activation function layer. Inadequate initialization can result in vanishing or exploding gradients. In the first method, we propose an optimization strategy for initializing weights and bias in the convolutional block to ameliorate the learning of features so that each neuron learns as much as possible. Next, the activation function normalizes the output. We implement the convolutional block hyperparameters optimization by setting the convolutional weights initialization to constant, the bias initialization to zero and the Leaky ReLU activation function at the output. In the second method, we propose a new activation, ModReLU, in the activation layer of CNN. Additionally, we also introduce a Gaussian low pass filter to minimize image noise and improve the structures of images at distinct scales. Moreover, we add a pixel-domain-based color adjustment implementation to enhance the capacity of the proposed strategies. The proposed implementations handle better rotation, moving, occlusion and appearance change problems and improve tracking speed. Our experimental results clearly show a significant improvement in the overall performance compared to the original SiamFC tracker. The first proposed technique of this work surpasses the original fully convolutional Siamese networks (SiamFC) on the VOT 2016 dataset with an increase of 15.42% in precision, 16.79% in AUPC and 15.93% in IOU compared to the original SiamFC. Our second proposed technique also reveals remarkable advances over the original SiamFC with 18.07% precision increment, 17.01% AUPC improvement and an increase of 15.87% in IOU. We evaluate our methods on the Visual Object Tracking (VOT) Challenge 2016 dataset, and they both outperform the original SiamFC tracker performance and many other top performers.